Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix kafka avro wire format #2356

Merged
merged 9 commits into from
Oct 16, 2023
Merged

Fix kafka avro wire format #2356

merged 9 commits into from
Oct 16, 2023

Conversation

Licenser
Copy link
Member

@Licenser Licenser commented Oct 12, 2023 •
edited
Loading

Pull request

Description

  • remove schema registry preprocessor
  • remove index lookup and schema registry lookup in avro codec
  • add confluent_schema_registry codec for avro
  • move pre and post-processors to tremor-interceptor
  • add tremor-config

Checklist

  • The RFC, if required, has been submitted and approved
  • Any user-facing impact of the changes is reflected in docs.tremor.rs
  • The code is tested
  • Use of unsafe code is reasoned about in a comment
  • Update CHANGELOG.md appropriately, recording any changes, bug fixes, or other observable changes in behavior
  • The performance impact of the change is measured (see below)

Performance

@Licenser
Fix prefix decoding
70885fc 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
save
e258c76 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
Cleanup
d41a1e3 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
Extract pre/postprocessors
993ec29 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
Remove avro kafka schema registry leftovers
ed24a4e 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser Licenser force-pushed the fix-kafka-avro-wire-format branch 6 times, most recently from 0205203 to 6c4d872 Compare October 13, 2023 12:33
@codecov
Copy link

codecov bot commented Oct 13, 2023 •
edited
Loading

Codecov Report

Merging #2356 (2d1aa1a) into main (92b1f0d) will increase coverage by 0.01%.
The diff coverage is 87.46%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2356      +/-   ##
==========================================
+ Coverage   90.87%   90.88%   +0.01%     
==========================================
  Files         279      281       +2     
  Lines       59553    59675     +122     
==========================================
+ Hits        54117    54238     +121     
- Misses       5436     5437       +1
Flag Coverage Δ
unittests 90.88% <87.46%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/config.rs 98.35% <100.00%> (+0.04%) ⬆️
src/connectors/google.rs 88.70% <100.00%> (ø)
src/connectors/impls/bench.rs 86.78% <100.00%> (ø)
src/connectors/impls/cb.rs 90.70% <100.00%> (+0.16%) ⬆️
src/connectors/impls/clickhouse.rs 97.17% <100.00%> (ø)
src/connectors/impls/crononome.rs 95.60% <100.00%> (ø)
src/connectors/impls/dns/client.rs 82.56% <100.00%> (ø)
src/connectors/impls/elastic.rs 94.28% <100.00%> (ø)
src/connectors/impls/exit.rs 97.40% <100.00%> (ø)
src/connectors/impls/file.rs 91.79% <100.00%> (ø)
... and 99 more

... and 2 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92b1f0d...2d1aa1a. Read the comment docs.

@Licenser Licenser force-pushed the fix-kafka-avro-wire-format branch 3 times, most recently from b13ed01 to 9f55021 Compare October 13, 2023 13:32
@Licenser
Add changelog and cleanup
8a6f714 
Signed-off-by: Heinz N. Gies <[email protected]>
mfelsche
mfelsche previously approved these changes Oct 14, 2023
View reviewed changes
Copy link
Member

@mfelsche mfelsche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor typoes, otherwise 👍

tremor-codec/src/codec/avro.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/avro.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/kafka_schema_registry.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/kafka_schema_registry.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/kafka_schema_registry.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/avro.rs Outdated Show resolved Hide resolved
tremor-codec/src/lib.rs Outdated Show resolved Hide resolved
tremor-interceptor/Cargo.toml Outdated Show resolved Hide resolved
@Licenser @mfelsche
Apply suggestions from code review
aedc854 
Co-authored-by: Matthias Wahl <[email protected]>
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
include feedback
b753356 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
Copy link
Member Author

Licenser commented Oct 16, 2023 •
edited
Loading

All fixed :) thanks for the detailed review!

@Licenser
simd-json update
2d1aa1a 
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser Licenser merged commit 81897ea into main Oct 16, 2023
18 checks passed
@Licenser Licenser deleted the fix-kafka-avro-wire-format branch October 16, 2023 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mfelsche mfelsche mfelsche approved these changes

@darach darach Awaiting requested review from darach darach is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Licenser @mfelsche